#bash (2020-02)
A channel for bash scripting
2020-02-26
![Hemanth avatar](https://secure.gravatar.com/avatar/4230289cedd3353c68cde50d0663818b.jpg?s=72&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0014-72.png)
@Hemanth has joined the channel
![Adam Crews avatar](https://secure.gravatar.com/avatar/a89eb6a92f44dab8ce702283b7a38820.jpg?s=72&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0002-72.png)
@Adam Crews has joined the channel
![Erik Osterman (Cloud Posse) avatar](https://secure.gravatar.com/avatar/88c480d4f73b813904e00a5695a454cb.jpg?s=72&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0023-72.png)
@Erik Osterman (Cloud Posse) has joined the channel
![Tyrone Meijn avatar](https://secure.gravatar.com/avatar/60d52311d6f51d9f6beb1173c5b8e735.jpg?s=72&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0007-72.png)
@Tyrone Meijn has joined the channel
![Erik Osterman (Cloud Posse) avatar](https://secure.gravatar.com/avatar/88c480d4f73b813904e00a5695a454cb.jpg?s=72&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0023-72.png)
@Hemanth any burning bash questions?
![Jeremy G (Cloud Posse) avatar](https://avatars.slack-edge.com/2020-07-04/1229022582372_22757dbc9ef96d371614_72.jpg)
@Jeremy G (Cloud Posse) has joined the channel
![Andriy Knysh (Cloud Posse) avatar](https://avatars.slack-edge.com/2018-06-13/382332470551_54ed1a5d986e2068fd9c_72.jpg)
@Andriy Knysh (Cloud Posse) has joined the channel
![Ievgenii Shepeliuk avatar](https://secure.gravatar.com/avatar/0a09fc8daf94faf9a7d7a930142e2e60.jpg?s=72&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0014-72.png)
@Ievgenii Shepeliuk has joined the channel
![Hemanth avatar](https://secure.gravatar.com/avatar/4230289cedd3353c68cde50d0663818b.jpg?s=72&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0014-72.png)
@Erik Osterman (Cloud Posse) haha yes
![Hemanth avatar](https://secure.gravatar.com/avatar/4230289cedd3353c68cde50d0663818b.jpg?s=72&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0014-72.png)
I am trying to do some sorting stuff, my output is like (those are aws instance IDs)
Instance-BCD222
19.5140495
12.7219977
13.3188209
Instance-OIUQWE
Instance-QOWIEU
Instance-QOWIEU
Instance-BCD123
19.5140495
12.7219977
13.3188209
Instance-QOWIEU
Instance-QWOEIU
Instance-ASLDKJ
Instance-BCD231
19.5140495
12.7219977
13.3188209
I want to remove those instances Id’s in that list that doesn’t have numbers under them
![Hemanth avatar](https://secure.gravatar.com/avatar/4230289cedd3353c68cde50d0663818b.jpg?s=72&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0014-72.png)
Presently using with jq
for i in $(aws ec2 describe-instances | jq -r '.["Reservations"]|.[]|.Instances|.[]| .InstanceId' | sort -n)
do
echo "$i"
aws cloudwatch get-metric-statistics --metric-name CPUUtilization --start-time 2020-02-20T12:00:00 --end-time 2020-02-20T13:00:00 --period 60 --namespace AWS/EC2 --extended-statistics p80 --dimensions Name=InstanceId,Value=$i | jq -r '.Datapoints[]|select(.ExtendedStatistics.p80 >=3 and .ExtendedStatistics.p80 >=0 )| [.ExtendedStatistics.p80] | @tsv'
done
![Jeremy G (Cloud Posse) avatar](https://avatars.slack-edge.com/2020-07-04/1229022582372_22757dbc9ef96d371614_72.jpg)
Please describe the output you desire. I agree with Erik: you probably want to fix your queries rather than post-process them.
![Hemanth avatar](https://secure.gravatar.com/avatar/4230289cedd3353c68cde50d0663818b.jpg?s=72&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0014-72.png)
@Jeremy G (Cloud Posse) looking to eliminate those Instance-ID’s from output, that doesn’t show the numbers under them
Instance-BCD222
19.5140495
12.7219977
13.3188209
Instance-OIUQWE
Instance-QOWIEU
Instance-QOWIEU
Instance-BCD123
19.5140495
12.7219977
13.3188209
Instance-QOWIEU
Instance-QWOEIU
Instance-ASLDKJ
Instance-BCD231
19.5140495
12.7219977
13.3188209
![Jeremy G (Cloud Posse) avatar](https://avatars.slack-edge.com/2020-07-04/1229022582372_22757dbc9ef96d371614_72.jpg)
But what are you really trying to accomplish? Would it be OK if the output were
Instance-BCD222 19.5140495 12.7219977 13.3188209
Instance-BCD123 19.5140495 12.7219977 13.3188209
Instance-BCD231 19.5140495 12.7219977 13.3188209
![Hemanth avatar](https://secure.gravatar.com/avatar/4230289cedd3353c68cde50d0663818b.jpg?s=72&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0014-72.png)
To get list of the numbers i am trying with my conditions in the query with a instance id that i can associate, I believe the output you shared looks fine to me as well
![erik-stephens avatar](https://secure.gravatar.com/avatar/aa8044308a41350c418a1d736dc39662.jpg?s=72&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0017-72.png)
@Hemanth Perhaps this?
x=$(aws ...)
if test -n "$x"; then
echo $i
echo "$x"
fi
![Erik Osterman (Cloud Posse) avatar](https://secure.gravatar.com/avatar/88c480d4f73b813904e00a5695a454cb.jpg?s=72&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0023-72.png)
@Hemanth if you attach some JSON snippets that might help us with the jq
query
![Hemanth avatar](https://secure.gravatar.com/avatar/4230289cedd3353c68cde50d0663818b.jpg?s=72&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0014-72.png)
@Erik Osterman (Cloud Posse) JSON snippet
{
"Label": "CPUUtilization",
"Datapoints": [
{
"Timestamp": "2020-02-20T12:15:00Z",
"Unit": "Percent",
"ExtendedStatistics": {
"p80": 15.954233538159674
}
},
{
"Timestamp": "2020-02-20T12:50:00Z",
"Unit": "Percent",
"ExtendedStatistics": {
"p80": 15.954233538159674
}
},
{
"Timestamp": "2020-02-20T12:25:00Z",
"Unit": "Percent",
"ExtendedStatistics": {
"p80": 15.931399422129846
}
},
{
"Timestamp": "2020-02-20T12:30:00Z",
"Unit": "Percent",
"ExtendedStatistics": {
"p80": 15.86309297171495
}
},
{
"Timestamp": "2020-02-20T12:05:00Z",
"Unit": "Percent",
"ExtendedStatistics": {
"p80": 15.954233538159674
}
},
{
"Timestamp": "2020-02-20T12:40:00Z",
"Unit": "Percent",
"ExtendedStatistics": {
"p80": 15.931399422129846
}
},
{
"Timestamp": "2020-02-20T12:10:00Z",
"Unit": "Percent",
"ExtendedStatistics": {
"p80": 14.707647811929322
}
},
{
"Timestamp": "2020-02-20T12:45:00Z",
"Unit": "Percent",
"ExtendedStatistics": {
"p80": 15.931399422129846
}
},
{
"Timestamp": "2020-02-20T12:20:00Z",
"Unit": "Percent",
"ExtendedStatistics": {
"p80": 15.86309297171495
}
},
{
"Timestamp": "2020-02-20T12:55:00Z",
"Unit": "Percent",
"ExtendedStatistics": {
"p80": 15.931399422129846
}
},
{
"Timestamp": "2020-02-20T12:00:00Z",
"Unit": "Percent",
"ExtendedStatistics": {
"p80": 15.931399422129846
}
},
{
"Timestamp": "2020-02-20T12:35:00Z",
"Unit": "Percent",
"ExtendedStatistics": {
"p80": 15.86309297171495
}
}
]
}
![Hemanth avatar](https://secure.gravatar.com/avatar/4230289cedd3353c68cde50d0663818b.jpg?s=72&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0014-72.png)
The one’s(instances) that doesn’t have numbers underneath them looks like
{
"Label": "CPUUtilization",
"Datapoints": [
{
"Timestamp": "2020-02-20T12:15:00Z",
"Unit": "Percent",
"ExtendedStatistics": {
"p80": 0.0
}
},
{
"Timestamp": "2020-02-20T12:50:00Z",
"Unit": "Percent",
"ExtendedStatistics": {
"p80": 0.0
}
},
![erik-stephens avatar](https://secure.gravatar.com/avatar/aa8044308a41350c418a1d736dc39662.jpg?s=72&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0017-72.png)
@Hemanth 0.0 can’t be >= 3 and >= 0
![Jeremy G (Cloud Posse) avatar](https://avatars.slack-edge.com/2020-07-04/1229022582372_22757dbc9ef96d371614_72.jpg)
One main issue is that you should be outputting one array of Datapoints (on one line) per instance, and you are not. jq
is stream oriented, which is a different way of thinking. You will be helped by collecting the Datapoints into a single array, which is not what your jq
query is doing.
![Erik Osterman (Cloud Posse) avatar](https://secure.gravatar.com/avatar/88c480d4f73b813904e00a5695a454cb.jpg?s=72&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0023-72.png)
Oh, I think this is stemming from an earlier question that involved jq
?
![Hemanth avatar](https://secure.gravatar.com/avatar/4230289cedd3353c68cde50d0663818b.jpg?s=72&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0014-72.png)
yes
![Erik Osterman (Cloud Posse) avatar](https://secure.gravatar.com/avatar/88c480d4f73b813904e00a5695a454cb.jpg?s=72&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0023-72.png)
Since jq
is so powerful, better to share that part again.
![Hemanth avatar](https://secure.gravatar.com/avatar/4230289cedd3353c68cde50d0663818b.jpg?s=72&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0014-72.png)
trying to work my way through
![Hemanth avatar](https://secure.gravatar.com/avatar/4230289cedd3353c68cde50d0663818b.jpg?s=72&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0014-72.png)
sure
![Erik Osterman (Cloud Posse) avatar](https://secure.gravatar.com/avatar/88c480d4f73b813904e00a5695a454cb.jpg?s=72&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0023-72.png)
It may be easier to get the query right rather than offload it to bash.
![Hemanth avatar](https://secure.gravatar.com/avatar/4230289cedd3353c68cde50d0663818b.jpg?s=72&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0014-72.png)
yes i am amazed at jq
![Erik Osterman (Cloud Posse) avatar](https://secure.gravatar.com/avatar/88c480d4f73b813904e00a5695a454cb.jpg?s=72&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0023-72.png)
Yea, it’s awesome
![bradym avatar](https://avatars.slack-edge.com/2023-06-21/5464816405572_dd21bed1bf537acb6539_72.jpg)
@bradym has joined the channel
![Kendall Link avatar](https://secure.gravatar.com/avatar/ad72e9c79433730fbe2780a7e72bc490.jpg?s=72&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0023-72.png)
@Kendall Link has joined the channel
![Chris Fowles avatar](https://avatars.slack-edge.com/2019-10-08/789284772630_caabfcff3b09cf0455ee_72.jpg)
@Chris Fowles has joined the channel
![erik-stephens avatar](https://secure.gravatar.com/avatar/aa8044308a41350c418a1d736dc39662.jpg?s=72&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0017-72.png)
@erik-stephens has joined the channel
![osirisx.tls_cloudposs avatar](https://secure.gravatar.com/avatar/b97b6b71c57f3df72bb542194506637a.jpg?s=72&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0005-72.png)
@osirisx.tls_cloudposs has joined the channel
2020-02-27
![wattiez.morgan avatar](https://avatars.slack-edge.com/2023-10-20/6073093959986_e59384665a4ad7cf39b8_72.jpg)
@wattiez.morgan has joined the channel
2020-02-28
![Jeremy G (Cloud Posse) avatar](https://avatars.slack-edge.com/2020-07-04/1229022582372_22757dbc9ef96d371614_72.jpg)
@Hemanth try this:
for i in $(aws ec2 describe-instances | jq -r '.["Reservations"]|.[]|.Instances|.[]| .InstanceId' | sort -n); do echo "$i,$(aws cloudwatch get-metric-statistics --metric-name CPUUtilization --start-time 2020-02-20T12:00:00 --end-time 2020-02-20T13:00:00 --period 60 --namespace AWS/EC2 --extended-statistics p80 --dimensions Name=InstanceId,Value=$i | jq -r '[.Datapoints[]|select(.ExtendedStatistics.p80 >=0).ExtendedStatistics.p80] | @csv')" | grep -v ',$'; done
![bowtie](/assets/images/custom_emojis/bowtie.png)
![Hemanth avatar](https://secure.gravatar.com/avatar/4230289cedd3353c68cde50d0663818b.jpg?s=72&d=https%3A%2F%2Fa.slack-edge.com%2Fdf10d%2Fimg%2Favatars%2Fava_0014-72.png)
@Jeremy G (Cloud Posse) Thanks this is Neat