Because the SET clause expect to modify a property's value. Properties must belongs to a node or relation to be modify. In your case, name is not a property anymore but just a string, so it can't be set.
What you want to do it's keep your node to modify it's property.
MATCH (n)
SET n.name = rTrim(n.name)
Note : In this case as Neo4j will have to filter all your nodes anyway to figure out which one ends with a trailing space, so you can just apply rTrim to all of them. Might be good to double check the performance if it's a production project.
It's highly recommended to keep in lower case all your properties name.
You don't need to return anything when you WRITE something but you can still return count(*). Neo4j Desktop will provide you some hits about your query result anyway.
I initially tested in 4.0.4, then tested in 4.2.6, and the message is more informatvie as below. Perhaps it doesn't like "CASE" starts with a new cypher statement:
Invalid input 'CASE': expected
<EOF>
"RETURN"
"CREATE"
"DELETE"
"SET"
"REMOVE"
"DETACH"
"MATCH"
"WITH"
"UNWIND"
"USE"
"CALL"
"LOAD"
"FROM"
"FOREACH"
"WHERE"
"MERGE"
"OPTIONAL"
"ORDER"
"SKIP"
"LIMIT"
"UNION"
"," (line 3, column 1 (offset: 37))
"CASE N when NOT NULL THEN replace(n.name,'[','') ELSE n.name END as newName"
^
Your cypher is good. The last error message was produced from 4.2.6 version, and I think it's more informative. Probably it doesn't like CASE starts a new statement.