Consider using pattern comprehensions or subqueries to take the place of your OPTIONAL MATCH + collects. Also, do a PROFILE of your query and look at how the number of rows flow. Watch for spikes where the rows go multiplicative, those are likely areas in the query where you're getting a cartesian product.
Remember: operations yield rows, and operations execute per row, so the more rows you see, the more work is being done, and if rows are multiplying out in ways that may not make sense to you, it likely means that your OPTIONAL MATCHes are generating additional rows, and subsequent operations will execute upon them. Control that by using scoped aggregations (in subqueries) or using pattern comprehensions (if doing the equivalent of OPTIONAL MATCH ... WHERE ... collect(<results>))