Thank you @gstrauss! The answer IS to use proxy.server with map-urlpath. Where I was going wrong before was I had included proxy-header within the proxy.server directive. It should be separate.

My lighttpd.conf file now has:

proxy.server = ( "/flask" => ( ( "host" => "127.0.0.1", "port" => "8080", "check-local" => "disable" ) ))
proxy.header = ( "map-urlpath" => ( "/flask" => "/" ))

which works fine. Thank you.