Today a short post regarding webhooks on vManage. The actual setup is already described by Cisco and Suchandan Reddy (Technical Marketing Engineer - Cisco SD-WAN). I will mention that the setup of an e-mail server and address is no longer neccessary in versions > 20.3, however, the webhook server’s username and password should be filled in. I usually set it to ‘dummy’/‘dummy’ as most webservers will ignore this data if not required. In this post though, I want to focus on an issue I had on two vManage instances and I’ve tried four different software versions with all the same result:
- 20.3.4
- 20.6.3
- 20.8.1
- 20.9.1
Problem
In this case the webhook stopped working unexpectely for both vManage instances and it took us a while to notice that something was out of the ordinary. To test webhook configuration and alerting I usually resort to Webhook.site. By using a curl command in the vShell of vmanage, it’s easy to verify if the webhook is working.
Note that you use your personal webhook.site link as this automatically generated by the site and you’ll not be able to view the results when using my link.
vManage:~$ curl -v -X POST -H 'Content-type: application/json' https://webhook.site/131c7929-9ca4-4640-9d8a-261e7171cce7
* Trying 46.4.105.116:443...
* Connected to webhook.site (46.4.105.116) port 443 (#0)
* error reading ca cert file /etc/ssl/certs/ca-certificates.crt (Error in the time fields of certificate.)
* Closing connection 0
curl: (77) Problem with the SSL CA cert (path? access rights?)
It seems that there is an issue with the ca-certificates.crt file in the underlying operating system.
Diagnose & Fix
What I ended up doing was to download another certificate file, this one being from the Mozilla CA via curl.
vmanage-IRE:~$ curl https://curl.se/ca/cacert.pem --insecure > ca-certificates.crt
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 217k 100 217k 0 0 10.1M 0 --:--:-- --:--:-- --:--:-- 10.1M
This allows curl to fetch the cacert.pem file without checking the certificate on the curl.se website. This also was a problem apparently, so I set the --insecure flag.
Then test if this solves the problem by connecting again to the webhook server, this time by manually specifying the ca-certificates file by using the --cacert ca-certificates.crt flag.
vmanage-IRE:~# curl -v -X POST -H 'Content-type: application/json' https://webhook.site/d438e39e-eee9-4db1-a66f-0924447f2313 --cacert ca-certificates.crt
* Trying 46.4.105.116:443...
* Connected to webhook.site (46.4.105.116) port 443 (#0)
* found 140 certificates in ca-certificates.crt
* ALPN, offering http/1.1
* SSL connection using TLS1.2 / ECDHE_RSA_AES_256_GCM_SHA384
* server certificate verification OK
* server certificate status verification SKIPPED
* common name: webhook.site (matched)
* server certificate expiration date OK
* server certificate activation date OK
* certificate public key: RSA
* certificate version: #3
* subject: CN=webhook.site
* start date: Sun, 31 Jul 2022 23:09:19 GMT
* expire date: Sat, 29 Oct 2022 23:09:18 GMT
* issuer: C=US,O=Let's Encrypt,CN=R3
* ALPN, server did not agree to a protocol
> POST /d438e39e-eee9-4db1-a66f-0924447f2313 HTTP/1.1
> Host: webhook.site
> User-Agent: curl/7.69.1
> Accept: */*
> Content-type: application/json
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< Server: nginx
< Content-Type: text/plain; charset=UTF-8
< Transfer-Encoding: chunked
< Vary: Accept-Encoding
< X-Request-Id: 9090d1d6-3b04-427a-826a-8d4a28e5cc0d
< X-Token-Id: d438e39e-eee9-4db1-a66f-0924447f2313
< Cache-Control: no-cache, private
< Date: Mon, 03 Oct 2022 12:30:58 GMT
<
* Connection #0 to host webhook.site left intact
This proves that the ca-certificates.crt file is indeed the issue. You can verify all the CAs in the original CA file (or the one you just downloaded) to see if the CA is even present or not:
openssl crl2pkcs7 -nocrl -certfile ca-certificates.crt | openssl pkcs7 -print_certs -text -noout
To actually solve this issue, you will need the help of Cisco TAC. There needs to be a token generated and then a token response needs to be entered to be able to modify the /etc/ssl/certs directory.
tools consent-token shell request duration 1440
Please copy the challenge phrase appearing between the two
asterisk lines, and use at the Response generation site:
*****************************************************************************************
<token>
*****************************************************************************************
tools consent-token shell validate-response <response>
After you gained shell access, you can use the following commands to fix the issue:
vshell
cp /etc/ssl/certs/ca-certificates.crt /etc/ssl/certs/ca-certificates.crt.backup
cp ca-certificates.crt /etc/ssl/certs/ca-certificates.crt
Verify that the read properties are still okay:
vmanage-IRE:~# ls -al /etc/ssl/certs/ca-certificates.crt
-rw-r--r-- 1 root root 191724 Apr 17 20:55 /etc/ssl/certs/ca-certificates.crt
Verify the connectivity again with a test, this time without explicitely pointing to a ca-certificates.crt file:
curl -v -X POST -H 'Content-type: application/json' https://webhook.site/d438e39e-eee9-4db1-a66f-0924447f2313
The result should be similar to when specifying the ca-certificates.crt file. As a last test, you can set up a temporary webhook in the vmanage pointing to the test webhook link and perform an action on a test device that will trigger this webhook.
Conclusion
It seems that the built-in ca-certificates.crt file is no longer up to date. For both vManage instances, I noticed that the last change date was in April 2022 and the webhook alerting stopped working for both somewhere at the end of August 2022. Updating the vManage to a more recent version has not solved this problem either. I’ve filed this issue as a bug with TAC and hope that they can tackle the problem in a future software update bundle.
For now though, your only solution is to open a TAC case in order to get the correct privileges for this directory. The impact of this issue seems to be limited and mostly pertain to when the vManage needs to reach out to webservices, such as webhook servers. I’ve not verified vAnalytics for example, but this service could potentially be impacted, although this might be dependent on the signing CA. For our webhook server, as well as webhook.site, the signing CA is Let’s Encrypt. This CA did not directly appear in the original the CA certificates file, but the cross-signed CA does. It’s still unclear why the services stopped working as the ending dates seemed to be okay in the original file.