I believe the privacy concerns are made moot if all consumer level routers by default blocked incoming untracked connections and you need to poke holes in the firewall for the ports you need.
Having said that, even knowing the prefix it’s a huge address space to port scan through. So it’s pretty secure too with privacy extensions enabled.
But for sure the onus is on the router makers for now.
Actually how is your ISP giving out IPs to you? Mine uses IPv6 PD to give me a /48. And I then use SLAAC locally on the first /64 prefix on my LAN. Plus another /64 for VPN connections.
If you mean receiving RA/ND packets from your ISP (which are used to announce IPv6 prefixes) then you need to allow icmpv6 packets (if you don’t want to be able to be pinged, just block echo requests, ICMP in v4 and v6 carry important messages otherwise).
If your ISP uses DHCPv6 Prefix delegation you will need to allow packets to UDP port 546 and run a DHCPv6 client capable of handling PD messages.
If you have a fixed prefix, then you probably don’t need to use your ISPs SLAAC at all. You could just put your router on a fixed IP as <yourprefix>::1 and then have your router create RA/ND packets (radvd package in linux, not sure what it would be on pfsense) and assign IPs within your network that way.
If you have a dynamic prefix… It’s a problem I guess. But probably someone has done it and a google search will turn up how they handled it.
EDIT: Just clarified that the RA/ND packets advertise prefixes, not assign addresses.