Question, How do I know secretless broker is running properly?

Secretless Broker
1

Hello Cyberark secretless broker team. I’m not sure if this is the correct place to ask a question and get support, but going to give it a try. Please be patient with me as I can be a bit long winded.

So I’ve followed different guides and documents and think I have a correct setup. I have a docker image of secretless broker from dockerhub configured and started. The logs say it is listening.
logs:
////////////////////////////////////////////////////////////////////////////////////////////////
2020/02/04 23:19:53 Secretless v1.5.0-4dfeef3 starting up…
2020/02/04 23:19:53 Initializing health check on :5335…
2020/02/04 23:19:53 Initialization of health check done. You can access the endpoint at /live and /ready.
2020/02/04 23:19:53 [WARN] Plugin hashes were not provided - tampering will not be detectable!
2020/02/04 23:19:53 Trying to load ./secretless.yml…
2020/02/04 23:19:53 Configuration file ./secretless.yml loaded
2020/02/04 23:19:53 Registering reload signal listeners…
2020/02/04 23:19:53 [WARN] Starting TCP listener on 0.0.0.0:3306…
2020/02/04 23:19:53 [WARN] Starting TCP listener on 0.0.0.0:1433…

/////////////////// cursor just blinks after 2020/02/04 23:19:53 [WARN] Starting TCP listener on 0.0.0.0:1433… log.

I start two different dockized java spring boot web applications configured to access their respective databases via secretless broker, but both applications have connection errors at startup (connection refused). Both applications connect to their databases without secretless broker, but when I change the data source to localhost to access secretless broker I get connection errors, (spring.datasource.url=jdbc:sqlserver://localhost:1433;databaseName=“name”, and spring.datasource.url=jdbc:mysql://localhost:3306/“tableName”). I suspect that the issue might be with secretless broker but I don’t have any logs on the secretless broker side to start troubleshooting. I get error logs from the applications, but no new logs from secretless broker. I checked logs on webpage https://docs.conjur.org/Latest/en/Content/References/connectors/scl_ServiceConnectors_troubleshooting.htm?TocPath=Fundamentals|Secretless%20Pattern|Service%20Connectors|_____4 and I don’t see any logs that match mine. I also don’t see an example of logs that display a good running version of secretless broker.

What logs should be outputted by secretless broker if secretless broker is running correctly? Do you have any suggestions on where to start looking for the error or what the error might be?

The following are more details on my setup. I have minikube version v1.6.1 running in a VM virtualBox which is on my windows 10 computer. As previously mentioned within minikube, I have two docker containerized java spring boot web applications along with a docker container of secretless broker. The secretless.yml file being used by secretless broker is as follows, (some real data has been removed and replace with generic data placed within " " ):

version: “2”
services:
“app1”:
connector: mysql
listenOn: tcp://0.0.0.0:3306
credentials:
host: jdbc:mysql://“workingEndpoingToDB”
port: 3306
username: “username”
password:
from: conjur
get: secretlessBrokerSecrets/“DBPassword”
sslmode: disable

“app2”:
connector: mssql
listenOn: tcp://0.0.0.0:1433
credentials:
host: jdbc:sqlserver://“workingEndPointToDB”:1433;databaseName=“databaseName”
port: 1433
username: “userName”
password:
from: conjur
get: secretlessBrokerSecrets/“DBPassword”

I am using Conjur OSS as the secrets management tool. Without secretless broker, apps successfully retrieve secrets from Conjur OSS and access their databases.

I also have added the required environment variables to minikube so access to conjur OSS can be made.

Thank you for reading and I look forward to your suggestions.

2

Hey @LouisLouisLouL - I’ll try to help out!

  • First thing that needs mentioning is hostname for MSSQL server (localhost) in your JDBC: this cannot be “localhost” since it has a special meaning in MSSQL land so it should be either 127.0.0.1 or ..
  • Secretless can dump a bit more info with a -debug flag
  • If you are able to reach the broker at the opened port, you should be able to see a log message dumped to stdout (not sure if you need a debug flag for that). If you don’t see those messages, it might be a connectivity issue.
  • I am a bit worried about a number of inlined quotes in the connection strings - I would remove those first to see if that’s the cause.
  • Ports in your configs for those two databases are superfluous - they already have those default values.
  • I don’t know if formatting of your secretless config is exact but it looks like indentation is all odd - it would be nice to see this in a code block to make sure it’s formatted correctly.

For reference since it might be useful, this is an example of a working broker log with debugging on connecting to an mssql DB:

go run cmd/secretless-broker/main.go -f ~/checkout/tmp/broker-mssql/secretless.yml -debug                                                                                                                     
2020/01/23 10:34:42 Secretless v1.4.2-dev starting up...                                                                                                                                                                                                            
2020/01/23 10:34:42 Initializing health check on :5335...                                                                         
2020/01/23 10:34:42 Initialization of health check done. You can access the endpoint at `/live` and `/ready`.                                                                                                                                                       
2020/01/23 10:34:42 [WARN]  Plugin directory '/usr/local/lib/secretless' not found. Ignoring external plugins...                                                                                                                                                    
2020/01/23 10:34:42 Trying to load configuration file: /Users/foo/checkout/tmp/broker-mssql/secretless.yml                                                                                                                                                       
2020/01/23 10:34:42 [INFO]  Waiting for new configuration...                                                                                                                                                                                                        
2020/01/23 10:34:42 [DEBUG] Got new configuration                                                                                                                                                                                                                   
2020/01/23 10:34:42 [INFO]  Validating config against available plugins: ssh,ssh-agent,pg,mysql,mssql,aws,basic_auth,conjur,generic_http                                                                                                                            
2020/01/23 10:34:42 Registering reload signal listeners...                                                                                                                                                                                                          
2020/01/23 10:34:42 [WARN]  Starting TCP listener on 0.0.0.0:1433...                                                                                                                                                                                                
2020/01/23 10:34:42 [INFO]  mssql: Starting service                                                                                                                                                                                                                 
2020/01/23 10:34:42 [INFO]  Waiting for new configuration...                                                                                                                                                                                                        
2020/01/23 10:35:18 Instantiating provider 'literal'                                                                                                                                                                                                                
2020/01/23 10:35:18 [INFO]  mssql: New connection on 192.168.44.123:1433.                                                                                                                                                                                            
2020/01/23 10:35:29 [INFO]  mssql: New connection on 192.168.44.123:1433.

Let me know if this helps :slight_smile:

1 Like
3

@sgnn7 I think the target database is MySQL and not MSSQL (eg SQL Server). I don’t think that changes any of your recommendations, but just wanted to raise that in case your advice would be different :slight_smile:

4

@izgerij I think he’s trying to use both:

1 Like
5

As a sidenote, @LouisLouisLouL - you may find this post helpful. It gives advice about how to format code blocks in discourse, which would help make the log output and config YAML in your original post more readable :slight_smile:

Please let us know if the options @sgnn7 suggested make sense, and if you have any further questions!

1 Like
6

Thank you everyone for the suggestions. They make sense, good suggestions. I made the changes suggested, 127.0.0.1 as endpoint and not localhost in my application.properties file of my test apps. I also removed the port numbers on both the application.properties file and secretless.yml file. Ran secretless broker with debug flag. I see a few more logs. Unfortunately, I still have the same issues. I think I have two overall issues. One issue is that my web applications don’t connect to secretless broker. The other issue is that secretless broker probably doesn’t connect to the databases. I think connectivity issues between docker containers is probably out of scope for this forum, so I’ll look to docker forums on testing connectivity between containers and the sharing of a localhost. I do have more questions on troubleshooting and testing secretless broker and how it works. I am running secretless broker with the -debug flag and my logs match up with the sample logs provided except for the last 3 logs that display a new connection is made and provides an IP address and port number. When secretless broker boots up does it open connections to all services listed in the secretless.yml file or does it wait to receive a message on an incoming port and at that time open a connection to the needed service? The logs make me think all connections are made at bootup but was hoping someone could verify my assumption. If secretless broker opens all connections when it starts up, then I can troubleshoot secretless broker making a connection with the services independent of web applications connecting to secretless broker. Another question I had on secretless broker for troubleshooting purposes was on the behavior of secretless broker if all the credentials in the secretless.yml file are hardcoded and no secrets management mechanism is used. If all credentials are hardcoded, will secretless broker bypass all code to request a secret and establish a connection or will this cause some unknow behavior? I tried it out and I get the same log outputs. My assumption was that it would make a connection since it had all the credentials needed, but I get the same logs.

Screen shot of my secretless.yml file:

screenShotOfSecretlessyml
screenShotOfSecretlessyml1598×876 13 KB

My logs from secretless:
logsFromSecretless02_06_20.txt (1.0 KB)

endpoints from application.properties files:
spring.datasource.url=jdbc:mysql://127.0.0.1/petclinic
spring.datasource.url=jdbc:sqlserver://127.0.0.1;databaseName=name

7

Hello again @LouisLouisLouL. I will answer some of your questions, and hopefully @sgnn7 can continue to dig deeper into some of the others.

  • Secretless does support explicitly specifying the credential values in the secretless.yml - doing this would mean your credentials section of the config looks something like:
    credentials:
  host: mysqldb.us-east-2.rds.amazonaws.com
  port: 3306
  username: my-user
  password: topsecretpassword
  • You don’t need to include the jdbc:mysql:// prefix in your Secretless config - you can see an example in our MSSQL integration tests here where the host is referring to a specific container by its name
  • Secretless does not open any connections on startup, it only opens connections when a new connection request comes in from a client. In this way, Secretless ensures it’s always using the latest credential values from your secret store (if you’re using one) when it opens a new connection

I am wondering if just changing your secretless.yml to remove the jdbc:[db type]:// prefixes from the host entries would do the trick - please try that out and report back :slight_smile:

1 Like
8

Thank you again for the suggestions. After clearing up some connectivity issues between my test applications and the secretless broker docker container, I implemented the suggested changes to my secretless.yml file and I was able to see that my test application connected to secretless broker and then secretless broker connected to the backend database. So to me this means secretless broker is up and running as it should. My test application ran, I saw good logs. Thank you very much for the support. The secretless.yml file I used had the credentials for the database hardcoded. I do have other questions related to secretless broker, but I’ll open another questions for those. Thank you again for the help on understanding when secretless broker is running, how to setup the secretless.yml file, and what logs to see.

3 Likes
9

This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.