#bash (2020-02)

A channel for bash scripting

2020-02-28

Jeremy (Cloud Posse) avatar
Jeremy (Cloud Posse)

@Hemanth try this:

for i in $(aws ec2 describe-instances  | jq -r '.["Reservations"]|.[]|.Instances|.[]| .InstanceId' | sort -n); do echo "$i,$(aws cloudwatch get-metric-statistics --metric-name CPUUtilization --start-time 2020-02-20T12:00:00 --end-time 2020-02-20T13:00:00 --period 60 --namespace AWS/EC2  --extended-statistics p80 --dimensions Name=InstanceId,Value=$i | jq -r '[.Datapoints[]|select(.ExtendedStatistics.p80 >=0).ExtendedStatistics.p80] | @csv')" | grep -v ',$'; done
bowtie1
Hemanth avatar
Hemanth

@Jeremy (Cloud Posse) Thanks this is Neat

2020-02-27

wattiez.morgan avatar
wattiez.morgan
12:13:30 PM

@wattiez.morgan has joined the channel

2020-02-26

Hemanth avatar
Hemanth
06:01:16 PM

@Hemanth has joined the channel

Hemanth avatar
Hemanth
06:01:17 PM

@Hemanth set the channel description: A channel for bash scripting

1
Adam Crews avatar
Adam Crews
06:02:37 PM

@Adam Crews has joined the channel

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)
06:39:55 PM

@Erik Osterman (Cloud Posse) has joined the channel

Tyrone Meijn avatar
Tyrone Meijn
06:51:22 PM

@Tyrone Meijn has joined the channel

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

@Hemanth any burning bash questions?

Jeremy (Cloud Posse) avatar
Jeremy (Cloud Posse)
07:06:36 PM

@Jeremy (Cloud Posse) has joined the channel

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)
07:06:36 PM

@Andriy Knysh (Cloud Posse) has joined the channel

Ievgenii Shepeliuk avatar
Ievgenii Shepeliuk
07:10:10 PM

@Ievgenii Shepeliuk has joined the channel

Hemanth avatar
Hemanth

@Erik Osterman (Cloud Posse) haha yes

Hemanth avatar
Hemanth

I am trying to do some sorting stuff, my output is like (those are aws instance IDs)

Instance-BCD222
19.5140495 
12.7219977 
13.3188209
Instance-OIUQWE
Instance-QOWIEU
Instance-QOWIEU
Instance-BCD123
19.5140495 
12.7219977 
13.3188209
Instance-QOWIEU
Instance-QWOEIU
Instance-ASLDKJ
Instance-BCD231
19.5140495 
12.7219977 
13.3188209

I want to remove those instances Id’s in that list that doesn’t have numbers under them

Hemanth avatar
Hemanth

Presently using with jq

for i in $(aws ec2 describe-instances  | jq -r '.["Reservations"]|.[]|.Instances|.[]| .InstanceId' | sort -n)
do
echo "$i"
aws cloudwatch get-metric-statistics --metric-name CPUUtilization --start-time 2020-02-20T12:00:00 --end-time 2020-02-20T13:00:00 --period 60 --namespace AWS/EC2  --extended-statistics p80 --dimensions Name=InstanceId,Value=$i | jq -r '.Datapoints[]|select(.ExtendedStatistics.p80 >=3 and .ExtendedStatistics.p80 >=0 )| [.ExtendedStatistics.p80] | @tsv'
done
Jeremy (Cloud Posse) avatar
Jeremy (Cloud Posse)

Please describe the output you desire. I agree with Erik: you probably want to fix your queries rather than post-process them.

Hemanth avatar
Hemanth

@Jeremy (Cloud Posse) looking to eliminate those Instance-ID’s from output, that doesn’t show the numbers under them

Instance-BCD222
19.5140495 
12.7219977 
13.3188209
Instance-OIUQWE
Instance-QOWIEU
Instance-QOWIEU
Instance-BCD123
19.5140495 
12.7219977 
13.3188209
Instance-QOWIEU
Instance-QWOEIU
Instance-ASLDKJ
Instance-BCD231
19.5140495 
12.7219977 
13.3188209
Jeremy (Cloud Posse) avatar
Jeremy (Cloud Posse)

But what are you really trying to accomplish? Would it be OK if the output were

Instance-BCD222 19.5140495 12.7219977 13.3188209
Instance-BCD123 19.5140495 12.7219977 13.3188209
Instance-BCD231 19.5140495 12.7219977 13.3188209
Hemanth avatar
Hemanth

To get list of the numbers i am trying with my conditions in the query with a instance id that i can associate, I believe the output you shared looks fine to me as well

erik-stephens avatar
erik-stephens

@Hemanth Perhaps this?

x=$(aws ...)
if test -n "$x"; then
  echo $i
  echo "$x"
fi
1
Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

@Hemanth if you attach some JSON snippets that might help us with the jq query

Hemanth avatar
Hemanth

@Erik Osterman (Cloud Posse) JSON snippet

{
    "Label": "CPUUtilization",
    "Datapoints": [
        {
            "Timestamp": "2020-02-20T12:15:00Z",
            "Unit": "Percent",
            "ExtendedStatistics": {
                "p80": 15.954233538159674
            }
        },
        {
            "Timestamp": "2020-02-20T12:50:00Z",
            "Unit": "Percent",
            "ExtendedStatistics": {
                "p80": 15.954233538159674
            }
        },
        {
            "Timestamp": "2020-02-20T12:25:00Z",
            "Unit": "Percent",
            "ExtendedStatistics": {
                "p80": 15.931399422129846
            }
        },
        {
            "Timestamp": "2020-02-20T12:30:00Z",
            "Unit": "Percent",
            "ExtendedStatistics": {
                "p80": 15.86309297171495
            }
        },
        {
            "Timestamp": "2020-02-20T12:05:00Z",
            "Unit": "Percent",
            "ExtendedStatistics": {
                "p80": 15.954233538159674
            }
        },
        {
            "Timestamp": "2020-02-20T12:40:00Z",
            "Unit": "Percent",
            "ExtendedStatistics": {
                "p80": 15.931399422129846
            }
        },
        {
            "Timestamp": "2020-02-20T12:10:00Z",
            "Unit": "Percent",
            "ExtendedStatistics": {
                "p80": 14.707647811929322
            }
        },
        {
            "Timestamp": "2020-02-20T12:45:00Z",
            "Unit": "Percent",
            "ExtendedStatistics": {
                "p80": 15.931399422129846
            }
        },
        {
            "Timestamp": "2020-02-20T12:20:00Z",
            "Unit": "Percent",
            "ExtendedStatistics": {
                "p80": 15.86309297171495
            }
        },
        {
            "Timestamp": "2020-02-20T12:55:00Z",
            "Unit": "Percent",
            "ExtendedStatistics": {
                "p80": 15.931399422129846
            }
        },
        {
            "Timestamp": "2020-02-20T12:00:00Z",
            "Unit": "Percent",
            "ExtendedStatistics": {
                "p80": 15.931399422129846
            }
        },
        {
            "Timestamp": "2020-02-20T12:35:00Z",
            "Unit": "Percent",
            "ExtendedStatistics": {
                "p80": 15.86309297171495
            }
        }
    ]
}
Hemanth avatar
Hemanth

The one’s(instances) that doesn’t have numbers underneath them looks like

{
    "Label": "CPUUtilization",
    "Datapoints": [
        {
            "Timestamp": "2020-02-20T12:15:00Z",
            "Unit": "Percent",
            "ExtendedStatistics": {
                "p80": 0.0
            }
        },
        {
            "Timestamp": "2020-02-20T12:50:00Z",
            "Unit": "Percent",
            "ExtendedStatistics": {
                "p80": 0.0
            }
        },
erik-stephens avatar
erik-stephens

@Hemanth 0.0 can’t be >= 3 and >= 0

Jeremy (Cloud Posse) avatar
Jeremy (Cloud Posse)

One main issue is that you should be outputting one array of Datapoints (on one line) per instance, and you are not. jq is stream oriented, which is a different way of thinking. You will be helped by collecting the Datapoints into a single array, which is not what your jq query is doing. See my suggestion here: https://sweetops.slack.com/archives/CUJFKCUFJ/p1582916079012800

@Hemanth try this:

for i in $(aws ec2 describe-instances  | jq -r '.["Reservations"]|.[]|.Instances|.[]| .InstanceId' | sort -n); do echo "$i,$(aws cloudwatch get-metric-statistics --metric-name CPUUtilization --start-time 2020-02-20T12:00:00 --end-time 2020-02-20T13:00:00 --period 60 --namespace AWS/EC2  --extended-statistics p80 --dimensions Name=InstanceId,Value=$i | jq -r '[.Datapoints[]|select(.ExtendedStatistics.p80 >=0).ExtendedStatistics.p80] | @csv')" | grep -v ',$'; done
Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Oh, I think this is stemming from an earlier question that involved jq?

Hemanth avatar
Hemanth

yes

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Since jq is so powerful, better to share that part again.

Hemanth avatar
Hemanth

trying to work my way through

Hemanth avatar
Hemanth

sure

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

It may be easier to get the query right rather than offload it to bash.

Hemanth avatar
Hemanth

yes i am amazed at jq

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Yea, it’s awesome

bradym avatar
bradym
07:17:15 PM

@bradym has joined the channel

Kendall Link avatar
Kendall Link
08:16:37 PM

@Kendall Link has joined the channel

Chris Fowles avatar
Chris Fowles
09:15:29 PM

@Chris Fowles has joined the channel

erik-stephens avatar
erik-stephens
12:39:09 AM

@erik-stephens has joined the channel

osirisx.tls_cloudposs avatar
osirisx.tls_cloudposs
07:15:53 AM

@osirisx.tls_cloudposs has joined the channel

    keyboard_arrow_up