The UPS also reports battery voltage, but that can only be viewed via apcupsd apcaccess or HWMonitor in Windows. HWMonitor gives faster updates, but it can cause blue screen Windows crashes, even after quitting the program, so I recommend apcupsd. Both programs showed voltage fluctuations, from below 11.1V to 13.3V.
The real battery voltage, was 13.60V with the UPS off, and 13.55V with the UPS on. So, the battery and the charger were working properly, and the UPS was measuring an incorrect voltage. The UPS was functional, though the incorrectly low voltage measurements could lead to a short runtime.
I removed the circuit board from the UPS, and only powered it via the battery connections using a bench supply. This is a safe method for testing. The board does not have any high voltage capacitors, and it cannot produce high voltage without connection to the large transformer mounted elsewhere in the UPS case. The small transformer on the board is only used for powering the UPS circuit from AC and charging the battery.
Microchip PIC16C745. It has an analog to digital converter, so it's reasonable to suspect that the battery is measured that way. According to the data sheet, only 5 pins are usable as ADC channels. By varying the bench supply voltage while measuring voltage at those pins I saw that pin 3 was probably used to measure battery voltage. R15 and R2 form a voltage divider, and C18 is used to smooth the measured voltage. Fluctuations were also happening on the other end of R15, which is the positive "12V" supply to lots of things on the board. The supply comes from the positive battery terminal via the power switch and one of the transistors above the microcontroller.
Although it is a latching pushbutton, it works on the principle of a slide switch. It is DPDT, but is used as an SPST switch with the poles connected in parallel. The switch is hard to unsolder due to all the pins, but it is very easy to disassemble by prying the sides to release the bottom with the contacts. Be careful with the sliding contacts. They are tiny, light, delicate, and nothing is holding them in place. I cleaned them by passing paper moistened in isopropyl alcohol inside them. I also cleaned the stationary contacts that way, and cleaned them from the outside with a pen eraser. After that, the resistance was less than 0.1 Ω and the problem is fixed. I hope it stays fixed.
Further comments on the APC Back-UPS ES 500 BE500U-CNVoltage at the battery terminals goes above 14V when the UPS is AC powered and turned off, and no battery is connected. This would be bad for a battery, but even a 10 kΩ resistor is enough to bring the voltage down within acceptable levels. I don't think this is a problem, because there's probably more than 1.4 mA flowing into a fully charged battery.
Note the big metal blocks on the circuit board near the power switch. Those are heat sinks for the inverter MOSFETs. A normal heat sink would have fins to radiate heat. The metal blocks mainly provide heat capacity. The run time of the UPS is limited by battery capacity and charging speed. It is okay if those blocks need an hour to cool down, because battery charging takes even longer. Those heat sinks would need to be replaced if you want to use a much higher capacity battery or use the UPS continuously as an inverter.