DSpace 5.x comes with a very old PostgreSQL JDBC driver (v9.1-901-1.jdbc4): https://github.com/DSpace/DSpace/blob/dspace-5_x/pom.xml#L1162
If you install DSpace 5 with more recent versions of Postgres (especially Postgres 10), Groups can no longer be created. Specifically a NullPointerException occurs when DSpace attempts to create the default Administrator/Anonymous groups.
The fix is to simply update to the newest version of the PostgreSQL driver (currently 42.2.1). As noted in its documentation, this new driver works for Postgres 8 and above. See: https://github.com/pgjdbc/pgjdbc#supported-postgresql-and-java-versions
A sample of configuring this new driver in Maven can be found at: https://github.com/pgjdbc/pgjdbc#maven-central
I've not tested whether DSpace 6 would also be affected by similar issues. It also uses a very old JDBC driver (v9.4.1211): https://github.com/DSpace/DSpace/blob/dspace-6_x/pom.xml#L27