목록IT/Python (6)

진지한 개발자

unzip

import json, gzip import shutil tdc_gz = '/data001/jupyter-docker/jooeun.kim/mon/raw/integration-parser-v1_9_9_027fc400-f327-8e67-2fee-ac46c2111cda_6992502d-31eb-9081-c152-70de02e65ce1_gz-0.gz' # with fs.open(tdc_gz) as s3fp: with gzip.open(tdc_gz, 'rb') as f_in: with open('tdc.json', 'wb') as f_out: shutil.copyfileobj(f_in, f_out)
IT/Python 2023. 8. 26. 19:20
S3 file 지우기

aws s3 filter 조건의 file 지우기 import boto3 sess = boto3.Session() s3 = sess.resource('s3') operation_bucket = s3.Bucket('MY_BUCKET') for prd in prd_list: file_list_to_delete = [] for file_key in operation_bucket.objects.filter(Prefix=f'MY_PREFIX/product={prd}'): # print(file) if 'some_condition' in str(file_key): file_list_to_delete.append(file_key) s3.Object('MY_BUCKET', file_key._key).delete() pr..
IT/Python 2023. 6. 29. 08:41
특정 이름의 S3 파일을 지우고 싶을 때

특정 이름을 가진 S3 파일 리스트 필터링 import boto3 s3 = boto3.resource('s3') bucket = s3.Bucket('bucket-name') for file_key in file_list: print(file_key) s3.Object('s3-an2-op-hlp-etl', file_key).delete() 필터링 된 S3 리스트 내 파일 삭제 import boto3 sess = boto3.Session() s3 = sess.resource('s3') operation_bucket = s3.Bucket('bucket-name') file_list = [] for file in operation_bucket.objects.filter(Prefix=f'test_prefix/')..
IT/Python 2023. 6. 9. 15:37
Convert GroupBy Series to DataFrame

# Below are a quick example # Example 1: Convert groupby Series # Using groupby() & count() on multiple column grouped_ser = df.groupby(['Courses', 'Duration'])['Fee'].count() # Example 2: Convert groupby object to DataFrame grouped_df = grouped_ser.reset_index() # Example 3: Use the as_index attribute to get groupby DataFrame grouped_df = df.groupby(['Courses', 'Duration'], as_index = False)['F..
IT/Python 2023. 6. 5. 15:41
pip 설치 시 SSLError 오류 해결방법 (SSLCertVerificationError)

pip install # 'SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1045)'))' 해결 방법 pip --trusted-host pypi.org --trusted-host files.pythonhosted.org install
IT/Python 2023. 5. 22. 15:18
Python 유용한 코드

타입 변환 df['column_a'].str.slice(start=13, stop=15) # column_a : df["time"] = df["timestamp"].date # time : df["time"] = df["timestamp"].time from datetime import datetime df['timestamp'] = df['timestamp'].apply(lambda x: datetime.strptime(x, '%Y-%m-%d %H:%M:%S.%f')) df['Dates'] = pd.to_datetime(df['date']).dt.date df['Time'] = pd.to_datetime(df['date']).dt.time 파일 읽기 # !pip install awswrangler==2..
IT/Python 2023. 5. 20. 11:13
이전 Prev 1 Next 다음