#bash (2020-02)
A channel for bash scripting
2020-02-26
@Hemanth has joined the channel
set the channel description: A channel for bash scripting
1
@Adam Crews has joined the channel
@Erik Osterman (Cloud Posse) has joined the channel
@Tyrone Meijn has joined the channel
@Hemanth any burning bash questions?
@Jeremy G (Cloud Posse) has joined the channel
@Andriy Knysh (Cloud Posse) has joined the channel
@Ievgenii Shepeliuk has joined the channel
@Erik Osterman (Cloud Posse) haha yes
I am trying to do some sorting stuff, my output is like (those are aws instance IDs)
Instance-BCD222
19.5140495
12.7219977
13.3188209
Instance-OIUQWE
Instance-QOWIEU
Instance-QOWIEU
Instance-BCD123
19.5140495
12.7219977
13.3188209
Instance-QOWIEU
Instance-QWOEIU
Instance-ASLDKJ
Instance-BCD231
19.5140495
12.7219977
13.3188209
I want to remove those instances Id’s in that list that doesn’t have numbers under them
Presently using with jq
for i in $(aws ec2 describe-instances | jq -r '.["Reservations"]|.[]|.Instances|.[]| .InstanceId' | sort -n)
do
echo "$i"
aws cloudwatch get-metric-statistics --metric-name CPUUtilization --start-time 2020-02-20T12:00:00 --end-time 2020-02-20T13:00:00 --period 60 --namespace AWS/EC2 --extended-statistics p80 --dimensions Name=InstanceId,Value=$i | jq -r '.Datapoints[]|select(.ExtendedStatistics.p80 >=3 and .ExtendedStatistics.p80 >=0 )| [.ExtendedStatistics.p80] | @tsv'
done
Please describe the output you desire. I agree with Erik: you probably want to fix your queries rather than post-process them.
@Jeremy G (Cloud Posse) looking to eliminate those Instance-ID’s from output, that doesn’t show the numbers under them
Instance-BCD222
19.5140495
12.7219977
13.3188209
Instance-OIUQWE
Instance-QOWIEU
Instance-QOWIEU
Instance-BCD123
19.5140495
12.7219977
13.3188209
Instance-QOWIEU
Instance-QWOEIU
Instance-ASLDKJ
Instance-BCD231
19.5140495
12.7219977
13.3188209
But what are you really trying to accomplish? Would it be OK if the output were
Instance-BCD222 19.5140495 12.7219977 13.3188209
Instance-BCD123 19.5140495 12.7219977 13.3188209
Instance-BCD231 19.5140495 12.7219977 13.3188209
To get list of the numbers i am trying with my conditions in the query with a instance id that i can associate, I believe the output you shared looks fine to me as well
@Hemanth Perhaps this?
x=$(aws ...)
if test -n "$x"; then
echo $i
echo "$x"
fi
@Hemanth if you attach some JSON snippets that might help us with the jq query
@Erik Osterman (Cloud Posse) JSON snippet
{
"Label": "CPUUtilization",
"Datapoints": [
{
"Timestamp": "2020-02-20T12:15:00Z",
"Unit": "Percent",
"ExtendedStatistics": {
"p80": 15.954233538159674
}
},
{
"Timestamp": "2020-02-20T12:50:00Z",
"Unit": "Percent",
"ExtendedStatistics": {
"p80": 15.954233538159674
}
},
{
"Timestamp": "2020-02-20T12:25:00Z",
"Unit": "Percent",
"ExtendedStatistics": {
"p80": 15.931399422129846
}
},
{
"Timestamp": "2020-02-20T12:30:00Z",
"Unit": "Percent",
"ExtendedStatistics": {
"p80": 15.86309297171495
}
},
{
"Timestamp": "2020-02-20T12:05:00Z",
"Unit": "Percent",
"ExtendedStatistics": {
"p80": 15.954233538159674
}
},
{
"Timestamp": "2020-02-20T12:40:00Z",
"Unit": "Percent",
"ExtendedStatistics": {
"p80": 15.931399422129846
}
},
{
"Timestamp": "2020-02-20T12:10:00Z",
"Unit": "Percent",
"ExtendedStatistics": {
"p80": 14.707647811929322
}
},
{
"Timestamp": "2020-02-20T12:45:00Z",
"Unit": "Percent",
"ExtendedStatistics": {
"p80": 15.931399422129846
}
},
{
"Timestamp": "2020-02-20T12:20:00Z",
"Unit": "Percent",
"ExtendedStatistics": {
"p80": 15.86309297171495
}
},
{
"Timestamp": "2020-02-20T12:55:00Z",
"Unit": "Percent",
"ExtendedStatistics": {
"p80": 15.931399422129846
}
},
{
"Timestamp": "2020-02-20T12:00:00Z",
"Unit": "Percent",
"ExtendedStatistics": {
"p80": 15.931399422129846
}
},
{
"Timestamp": "2020-02-20T12:35:00Z",
"Unit": "Percent",
"ExtendedStatistics": {
"p80": 15.86309297171495
}
}
]
}
The one’s(instances) that doesn’t have numbers underneath them looks like
{
"Label": "CPUUtilization",
"Datapoints": [
{
"Timestamp": "2020-02-20T12:15:00Z",
"Unit": "Percent",
"ExtendedStatistics": {
"p80": 0.0
}
},
{
"Timestamp": "2020-02-20T12:50:00Z",
"Unit": "Percent",
"ExtendedStatistics": {
"p80": 0.0
}
},
@Hemanth 0.0 can’t be >= 3 and >= 0
One main issue is that you should be outputting one array of Datapoints (on one line) per instance, and you are not. jq is stream oriented, which is a different way of thinking. You will be helped by collecting the Datapoints into a single array, which is not what your jq query is doing.
Oh, I think this is stemming from an earlier question that involved jq?
yes
Since jq is so powerful, better to share that part again.
trying to work my way through
sure
It may be easier to get the query right rather than offload it to bash.
yes i am amazed at jq
Yea, it’s awesome
@bradym has joined the channel
@Kendall Link has joined the channel
@Chris Fowles has joined the channel
@erik-stephens has joined the channel
@osirisx.tls_cloudposs has joined the channel
2020-02-27
@wattiez.morgan has joined the channel
2020-02-28
@Hemanth try this:
for i in $(aws ec2 describe-instances | jq -r '.["Reservations"]|.[]|.Instances|.[]| .InstanceId' | sort -n); do echo "$i,$(aws cloudwatch get-metric-statistics --metric-name CPUUtilization --start-time 2020-02-20T12:00:00 --end-time 2020-02-20T13:00:00 --period 60 --namespace AWS/EC2 --extended-statistics p80 --dimensions Name=InstanceId,Value=$i | jq -r '[.Datapoints[]|select(.ExtendedStatistics.p80 >=0).ExtendedStatistics.p80] | @csv')" | grep -v ',$'; done
1
@Jeremy G (Cloud Posse) Thanks this is Neat