Outwashplain

Thats a good point, if they hold the stated equality then the reflection matrices probably have to look different from the transmission matrices.

It looks like a lot of the usual transmission matrices are projectors onto some basis element (either linear or circular polarization) which means that they should commute since each unique projector is in the null space of each other one. So if the reflection matrices are similar you could have a l...

You might also be able to work out the number of solutions? Like if the reflection matrices have a similar relation to the transmission matrices you can get a quadratic matrix equation for A in terms of L and B, and there are probably ways of working out when those have nonunique solutions and if so...

It seems like the family of matrices in question are pretty constrained. If you can find a parametrization of the set I think the algebra would get a lot easier.

If your goal is to get an estimate of where the valid configurations are its going to be hard to avoid taking a lot of samples, especially if you dont have any prior knowledge that the problem is linear or convex or something. If you just want to explore the space you can try something like bisectio...