We are facing issues getting AI Bridge to communicate with Milestone Management Client.
We have deployed the Helm chart and everything is successful, I even see it appear on the Processing Servers in Manageement client.
There are 2 issues:
Please find images below:
Hi Abaas,
This error message appears when the VMS is not being able to reach the Health endpoint.
As you mentioned, you are having an issue in getting the external IPs assigned to webservice and health pods. Which makes sense, that if there are no external IPs assigned to these pods then the Management client won’t be able to reach the health endpoint.
The Health endpoint is always exposed by the ingress controller, therefore, activating the debug mode or not should be the same (speaking of the health pod).
Would you mind sharing, if the same issue occurs when setting the debug mode to false?
And confirm if the issue occurs always or only when the debug mode is activated?
I will be waiting for your answer,
Daniel
Thanks for the reply! I tried both debug and not debug to check exactly that, and can confirm that it’s the exact same for both.
Hi again,
Can you as well. Check the `MIPtrace.log` file located at `[C:\ProgramData\Milestone\XProtect](file:C:/ProgramData/Milestone/XProtect) Management Client\Logs` and search for the produced error message and share here!
I will be waiting for your answer,
Daniel
Hi Daniel, I got acess to the machine and these are the only logs, there is nothinga about health endpoint in any of the logs. Same with Archive.
Hi Daniel, I will try and get this ASAP, as I don’t have 24/7 access to the client machine.
Hi Abaas,
Meanwhile you are trying to get the error message. Can you provide me with more details about the machines hosts. Can both hosts ping each other (the windows hosts running XProtect and the Linux hosts running AIB deployment)?
In the `values.yaml` file for AIBridge deployment (Helm configuration files). Can you check that the values you provided for the `externalIP` and `externalHostname` are correct and pointing the k8s cluster.
I will be waiting for your answer,
Daniel
Sure thing, here are the details:
Our Metallb system addresses: (we are using 10.128.4.221)
values.yaml
vms:
url: "http://milestonehusky"
bridge:
id: "12355b21-5a25-4a1d-b6d2-f6e02c9b95b4"
name: "EGX Cluster"
description: "Kubernetes cluster running the EGX software stack"
webpage: ""
gateway:
id: "1b80eaa0-203d-4dc0-ae3b-9bf4b85ec992"
version: "1.0.0"
replicas:
health: 1
connector: 1
streaming: 1
broker: 1
proxy: 1
webservice: 1
general:
tag: v1.7.2
debug: true
externalIP: "10.128.4.221"
externalHostname: "milestoneaibridge"
masterKey: "encryptionKeyExample"
ingress-nginx:
enabled: true
controller:
service:
externalIPs:
- "10.128.4.221"
I can ping `milestonehusky` from the network
There is also clearly a connection happening because the processing server sets up the application with the correct topics
These pods are getting IPs:
And these ones arent:
Hi Abaas,
Good morning.
Good. In regards of the log. Can you open the `MIPtrace.log` file and search for the error message.
For example, I took down the AIBridge service:In management client UI I could see the connection status message:
And by opening the logs I could see this error message right at the end of the file (it appears the end of the file because I opened the management client and navigated to processing server node):
This error message will let us know what was exactly the reason the plugin failed to connect with AIB health endpoint.
Now, I need that you confirm the following as well:
In one of the messages above you shared an terminal image from `milestoneaibridge` pinging `milestonehusky` but what about pinging the connection from the other direction. Something like, opening a terminal in the `milestonehusky` device and try to ping `milestoneaibridge` by name (please use the same name you assigned in the values.yaml file).
I will be waiting for your confirmation
Thank you
Daniel
Appreicate the quick response Daniel.
I should’ve mentioned I did do that, and searched for the error. But got nothing related to health endponits unfortunately. But I’ll have another look tonight when I get access again.
I’ll also make sure I can ping the milestoneaibridge endpoint as you mentioned.
Hi @Daniel Geruous
I got the logs
this looks like it’s to do with the connection not being successful.
But you mentioned to be able to ping the other way (i.e from milestonehusky to milestoneaibridge). As per my conversation with our network and K8s engineer, that wouldn’t ping anything back (and it didn’t) - because what are we pinging exactly? A pod? That isn’t expected to respond is it?
Please let me know if our reasoning is wrong there 
Hi Abaas,
From the logs we can understand that the machine where the VMS is running `milestonehusky` cannot resolve the hostname of the machine where the K8s cluster is running on.
I hope this diagram explains better the situation:
To clarify: When you ping the milestoneaibridge machine by name from within the milestonehusky server, you should ping the Linux server running the k8s cluster or for multi node cluster you should be able to ping the load balancer.
If that didn’t work then mind adding this extra host to the hosts file located at `[C:\Windows\System32\drivers\etc`](file:C:/Windows/System32/drivers/etc%60) with something similar to this
Why the issue should be solved by adding the extra host? Here is why
In the values.yaml you specify an externalHostname and externalIP. These values should be the same as your K8s cluster control plane node hostname and ip address in case the cluster is a single node. For multi node cluster the externalHostname and externalIP should be assigned with the hostname and ip address of the load balancer.
For AI Bridge, the ingress service is responsible of forwarding the requests coming to AIB to the right pod (including Health pod).
That means the Health pod and other pods does not require to be assigned a public IP address.
So the images you provided at the beginning are correct and the Health pod should not be assigned a public IP.
I hope this information helps you fix the problem and connect the systems,
Please, let me know if this didn’t work for you,
Have a nice day,
Daniel
@Daniel Geruous any idea here?
Thanks!
@Abaas Mahroos Have you read my answer from yesterday?
To help you I will need to know more about your infrastructure. I cannot do it without your collaboration.
All what I can ask you to do is (in case you are running a single node cluster):
If you cannot ping the cluster from the windows side then please mind adding the extra host to the windows machine so it can resolve the name of your Linux machine.
Another way to solve the issue is by assigning the `externalHostname` with the IP address as well.
Something like this:
general:
tag: v1.7.2
debug: true
externalIP: "10.128.4.221"
externalHostname: "10.128.4.221"
masterKey: "encryptionKeyExample"
Have a nice day,
Daniel
My apologies I did not see your reply (didn’t see that “read more” option).
I understand that I need to ping the LB and be able to get a response. I have been advised that pinging LB doesn’t return anything anyway so that isn’t a valid test (unless my K8s guys are wrong).
But regardless, should there not be an IP for the web service? Where I can run GraphQL and see the docs…etc?
Hi @Abaas Mahroos
In regards of the ping, I mentioned that you ping your Linux machine to see if it is reachable from the Windows machine network (I did not mention the LB pod).
The solution to your problem, as mentioned before in this thread and mentioned in the documentation shared with you on Nvidia’s private catalog, that in case you have a multi-node K8s cluster then use the IP address of your LB to fill the `externalHostname` and to the externalIP` in the values.yaml file:
general:
tag: v1.7.2
debug: true
externalIP: "<Your LB IP address>"
externalHostname: "<Your LB IP address>"
masterKey: "encryptionKeyExample"
Please, try that out and let me know of the results.
I will be waiting for your answer,
Daniel
Thanks Daniel, yes that’s exactly what we did.
We are able to ping our host machine (that is hosting K8s and ultimiatley AI Bridge) from our milestonehusky machine (that is hosting Milestone).
The externIP and externalHostname in the values.yaml is the IP address of our LB.
i.e. that is our LB address, we gave our setup (and the externalIP value) the IP: 10.129.4.221 (which is in our range above).
general:
tag: v1.7.2
debug: true
externalIP: "10.128.4.221"
externalHostname: "milestoneaibridge"
masterKey: "encryptionKeyExample"
The externalHostname: milestoneaibridge maps to 10.128.4.221.
Can the windows machine resolve the hostname milestoneaibridge to 10.128.4.221?
The externalHostname, can accept IP address as well. I would recommend you to use the IP address directly.
general:
tag: v1.7.2
debug: true
externalIP: "10.128.4.221"
externalHostname: "10.128.4.221"
masterKey: "encryptionKeyExample"
Can you please try that?
Hi @Abaas Mahroos,
I’m writing to you to let you know that I’m still working on the setting up the cluster with metallb, it is not ready yet.
I will need some information form you:
Can you please share with you metallb configuration (please if there any information that you don’t want to share just replace it with placeholder text).
Can you as well share what is the Kubernetes version you have?
You mentioned before that you were getting an error message when trying to set the IP address for the externalIP and externalHostname.
Can you please share a screenshot of the error message?
I will be waiting for your answer,
Daniel
