Neo4J HA Docker Cluster on ECS with Cloudformation Reference?

[arniesaha]

Hi Folks,

I've been developing a product with Neo4j over the last several months for which a single instance of Neo4j docker instance has sufficed my requirement.

I'm soon moving to production and I'm attempting to deploy a Neo4J docker container cluster on Amazon ECS with Cloudformation, and have been running into several issues.

Also, the reason why I also don't want to deploy the cluster AMI available on Amazon Marketplace is because it doesn't have availability in my required region. i.e ap-south-1

I've referenced something that looks like this: GitHub - arniesaha/neo4j-aws-ha-cluster: Neo4j Enterprise HA Cluster on AWS ECS

While, the docker image is generated and deployed on ECR as expected. I'm having issues deploying the cloudformation.yml using the image and either the Tasks under the service would crash or the deployment times-out and rolls back.

Details of the issue here: The security group 'sg-521f642f' does not exist (Service: AmazonAutoScaling; Status Code: 400; Error Code: ValidationError; Request ID: xxx) · Issue #1 · getsocial-rnd/neo4j-aws-ha-cluster · GitHub

Also, I would like to deploy with my existing volume from the single instance from my development along with the geospatial plugin.

Would there be a reference docker HA cluster implementation of Neo4j available with anyone?

Would highly appreciate any assistance with this!

Thanks,
Arnab

[david_allen]

It's OK if you'd prefer to use docker images. However if the issue is that you can't use the AMI because it's not available in your required region, it's possible to copy AMIs between regions so that it would be for you.

https://scrapoxy.readthedocs.io/en/master/standard/providers/awsec2/copy_ami_to_region/

I'm personally not familiar with this repo that you've forked as I haven't been using ECS. And it looks like the issue you're experiencing is buried pretty deep in the AWS specific bits. There is a reference docker HA cluster implementation of Neo4j that you can find running inside of kubernetes here: neo4j-google-k8s-marketplace/USER-GUIDE.md at 3.5 · neo-technology/neo4j-google-k8s-marketplace · GitHub

But the stack this uses is quite a bit different than what you're doing here with AWS and ECR.

You have a couple of options, depending on what you need that I can see. If AWS + ECR is best for you, then working with the maintainer of the repo you've forked is probably best. If you just need a cluster up and running quickly, Neo4j provides in the marketplace for all 3 major clouds an option to get you started. Most are VM based, but above I've linked a docker/kubernetes based approach as well.

Hosting Neo4j in the Cloud - Developer Guides

[arniesaha]

Hi David,

Thanks for the reply!

While, I'd prefer a docker cluster on AWS. But, even a VM approach is fine at the moment in interest of time to go live. Its easier to manage sheeps than pets ;)

But, I went ahead and gave this a try: AWS Marketplace: Neo4j Enterprise Causal Cluster

Which has a cloudformation template and service catalog options.

I chose the cloudformation method in a available zone, and tried to copy the available AMI to my required region. But, I run into this

Basically, it doesn't let me copy it.

And I'm not that well versed with GCP and Kubernetes and would like to keep our implementation on AWS due to business requirements.

Any more ideas?

Thanks,
Arnab

[david_allen]

Your best bet if you want to stick with Amazon is to use the marketplace entry:

This should require very little setup on your part.

AWS Marketplace: Neo4j Enterprise Causal Cluster

Unfortunately I can't provide support for the github repo you're using because I'm simply not familiar with it.

[arniesaha]

Hi David!

Sure, I understand. And likewise I mentioned I tried copying the official AMI to my required region and Amazon isn't allowing me to do so, like in the screenshot.

Will it be possible to make the image available in ap-south-1 region?

Regards,
Arnab

[david_allen]

Give me the neo4j version number you're trying to run, or the AMI ID of what you tried to copy and I can get you one in ap-south-1.

[arniesaha]

Hi David,

AMI ID: ami-0ae3b1104eed0d04c (available in ap-southeast-1)
Either 3.5.1 or 3.4.9 should work.

Thanks again,
Arnab

[david_allen]

AMI ami-0841505f29ee8c75f is neo4j 3.5.1 enterprise in ap-southeast-1, and should be available.

[arniesaha]

Thanks!

You mean ami-0841505f29ee8c75f should be available in ap-south-1 (Mumbai), right?

Cause it's probably still not available right now when I checked

[david_allen]

Shoot. Sorry about that. It's really easy for me to get the zone designators (ap-south-1) and the geo designators (Singapore) mixed up. In my previous message I got the AMI ID right but had copied it to the wrong region.

Try AMI ami-0284a2c822c6c3b9e

[arniesaha]

No worries! Thanks

The new AMI ami-0284a2c822c6c3b9e still doesn't reflect under ap-south-1. Perhaps, it takes a while to update on the marketplace?

I'll check back again in sometime and report back.

[david_allen]

Please post a screenshot of what you're seeing.

This is what I'm seeing.

[arniesaha]

@arniesaha

ami-0284a2c822c6c3b9e

This is what I see.

In your screenshot the visibility is set to private. Maybe you need to make it public for me to get access?

Thanks,
Arnab

[david_allen]

Arg, I have clearly not had enough coffee yet today!

It's just been made public.

[arniesaha]

Haha, I can understand.

I can see it now!

Thanks!

[arniesaha]

So I ran into another issue. I tried to use the ami mapping of ap-south-1 with the available cloudformation template available with AWS Marketplace: Neo4j Enterprise Causal Cluster

Since, ap-south-1 doesn't have a third availability zone like the other region, the template fails for subnet-3 e.g. that could be ap-southeast-1c

I tried some edits with the template but it might take longer to get it all wired up with 2AZs.

So, again in interest of time I wanted to launch the VMs to evaluate in ap-southeast-1

Everything went through fine. And I see the 3 nodes of EC2 created

And see a private VPC and couple of entries under route 53

The question is how can I access the neo4j browser? Don't see any load balancers created for the VPC or the Domain enteries?

Thanks,
Arnab

[david_allen]

No load balancers are created or necessary.

You can access neo4j browser on port 7473 of the public IP of any of the three machines you created.

Yes -- we usually deploy the AMIs by default to regions with a minimum of 3 AZs so that we can round-robin the machines around the AZs for high availability.

[arniesaha]

Got it.

So i tried accessing the 7473 on one of the deployed machines. e.g. 54.255.154.98 (in the screenshot below)

And the security rules like below

But the page doesn't seem come up

Anything im getting wrong with this?

Thanks,
Arnab

[david_allen]

This should be working, but it's hard to tell why it isn't from this. Did you make other changes to the CloudFormation that could affect routing, for example with internet gateways, subnets, etc?

If you ssh into the machine, can you from that machine access the service (cypher-shell -u neo4j -p password -a localhost)?

What does systemctl status neo4j say on the machine, and are there errors in debug.log?

[arniesaha]

Didn't change anything with the template. Infact launched it directly from the marketplace listing itself.

@david_allen

cypher-shell -u neo4j -p password -a localhost

Cypher shell command gives the above error.

While, systemctl status neo4j

Infact, also tried launching it another zone but running into similar behaviour. Will try once more though.

If you have any ideas do let me know.

Thanks,
Arnab